Skip to content

feat(rails): support for solid queue#2942

Open
solnic wants to merge 40 commits into
2933-active-job-tracing-specsfrom
2587-support-solid-queue
Open

feat(rails): support for solid queue#2942
solnic wants to merge 40 commits into
2933-active-job-tracing-specsfrom
2587-support-solid-queue

Conversation

@solnic

@solnic solnic commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

This adds support for SolideQueue by simply adding its spec suite that makes sure our common ActiveJob integration works with SolidQueue, including distributed tracing.

We may follow up with SQ-specific features added in separate PR(s).

⚠️ this builds on top of #2947 which should be reviewed/merged first.


Closes #2587

@solnic solnic linked an issue Apr 28, 2026 that may be closed by this pull request
@solnic solnic force-pushed the 2587-support-solid-queue branch from ba6dae4 to 0531f06 Compare April 30, 2026 13:23
@solnic solnic force-pushed the 2587-support-solid-queue branch from 396f231 to 22e181a Compare May 7, 2026 13:58
@solnic solnic changed the base branch from master to 2933-active-job-tracing-specs May 7, 2026 13:58
@solnic solnic force-pushed the 2933-active-job-tracing-specs branch 2 times, most recently from cafb0e5 to d61680a Compare May 12, 2026 08:14
@solnic solnic force-pushed the 2587-support-solid-queue branch from 22e181a to ad38485 Compare May 12, 2026 08:14
@solnic solnic force-pushed the 2933-active-job-tracing-specs branch 9 times, most recently from 826f69d to 1c0f001 Compare May 22, 2026 14:42
@solnic solnic force-pushed the 2587-support-solid-queue branch 2 times, most recently from 6322da1 to edaee65 Compare May 25, 2026 10:06
@solnic solnic force-pushed the 2587-support-solid-queue branch 2 times, most recently from edaee65 to 42e895a Compare June 1, 2026 10:16
@solnic solnic force-pushed the 2933-active-job-tracing-specs branch from 3eb29b0 to 6a62844 Compare June 9, 2026 11:09
@solnic solnic force-pushed the 2587-support-solid-queue branch from 42e895a to f1da26b Compare June 9, 2026 11:10
@solnic solnic force-pushed the 2933-active-job-tracing-specs branch from c9da666 to d072fd5 Compare June 12, 2026 09:36
@solnic solnic force-pushed the 2587-support-solid-queue branch 3 times, most recently from 951fdcb to d9baea2 Compare June 16, 2026 11:35
solnic and others added 4 commits June 22, 2026 08:02
Sets messaging.message.id, messaging.destination.name,
messaging.message.retry.count, and messaging.message.receive.latency
on the consumer transaction, mirroring sentry-sidekiq's middleware.

Adds an opt-in shared example that adapters can include to verify
the data fields are populated correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps ActiveJob enqueue with a `queue.publish` child span when an
active parent transaction exists, mirroring sentry-sidekiq's client
middleware. Uses the public `around_enqueue` callback so no new
ActiveJob monkey-patching is introduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the OpenTelemetry pattern (the only documented way to add
metadata to an ActiveJob payload — Rails has no public extension hook
for serialize/deserialize): prepends the existing ActiveJobExtensions
module with serialize/deserialize overrides that inject and recover
sentry-trace and baggage headers under a namespaced "_sentry" key,
wrapped in rescue blocks so a Sentry bug never breaks job execution.

Threads the deserialized headers into SentryReporter.record, which now
uses Sentry.continue_trace when present so the consumer transaction
shares the producer's trace_id and chains under the producer
queue.publish span.

Guards the around_enqueue producer-span registration against duplicate
registration (each Test::Application.define re-runs the railtie and
without idempotency this stacks dozens of nested queue.publish spans).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…er spec

The producer-span change makes ActiveStorage's internally-enqueued
AnalyzeJob emit an extra queue.publish span on the request transaction,
which the previous index-based span lookups did not anticipate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
solnic and others added 21 commits June 22, 2026 08:02
The harness embedded :test-adapter specifics — the Rails 5.2 payload-
preservation shim, the drain loop, and the enqueued-payload accessor.
It also reached past ActiveJob::TestHelper to set
ActiveJob::Base.queue_adapter directly, which conflicts with TestHelper's
own _test_adapter slot (TestHelper's before_setup runs outside our around
hook, so any direct assignment is silently shadowed).

Switch the harness to ActiveJob's official queue_adapter_for_test hook
and a small set of abstract methods (queue_adapter_for_test,
with_adapter_active, drain, last_enqueued_payload, boot_adapter,
reset_adapter) that adapter contexts implement. The :test-adapter
shared context now owns everything specific to TestAdapter — including
the Rails52FullPayloadTestAdapter shim and the drain loop. Subsequent
adapter backends (e.g. Sidekiq) can compose with the harness without
fighting it.

Generalises the one shared-example line that reached into the
TestAdapter shape (trace_propagation) via last_enqueued_payload.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…adapter

Runs the common ActiveJob spec suite end-to-end against
ActiveJob::QueueAdapters::SidekiqAdapter, driven by
Sidekiq::Testing.fake! (block form, public API) and Sidekiq::Job.drain_all.
Validates that the AJ tracing extension works as a generic, adapter-
agnostic instrumentation — independent of sentry-sidekiq's native
middleware.

The :sidekiq context plugs into the harness via queue_adapter_for_test
(installing a SidekiqAdapter instance through ActiveJob::TestHelper) and
with_adapter_active (wrapping example.run in Sidekiq::Testing.fake! so
fake mode is scoped per-example without touching global state).

The context deliberately does not load sentry-sidekiq: loading it would
install Sidekiq's client/server middleware globally and register
SidekiqAdapter in skippable_job_adapters, both of which would
short-circuit the AJ extension we're exercising.

Sidekiq becomes a sentry-rails dev dependency, gated on Rails version
(Sidekiq 7+ doesn't support Rails 5.2). The spec file and support file
no-op cleanly on older matrices where the gem isn't bundled.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drives the svelte-mini app to click a new "Trigger Job" button, which
fetches POST /jobs/sample on the rails-mini app. The browser SDK
propagates sentry-trace + baggage to the Rails request; the AJ
extension this branch adds emits a queue.publish span on the
http.server transaction at enqueue, and a queue.active_job consumer
transaction when the :async pool runs the job. The spec asserts all
three rails-side artifacts share one trace and are correctly linked
(sentry-trace header on the controller request, parent_span_id on the
consumer transaction, and matching messaging.* data on the producer
and consumer ends).

Polls the shared envelope log because :async runs the job on a
separate thread, so the HTTP response returns before the consumer
transaction is recorded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The harness was calling make_basic_app in its around-each block,
which creates a fresh Rails::Application subclass and runs every
initializer on each example. With 98 AJ examples that overhead dwarfed
the actual test work — and worse, it left behind state (Sidekiq's
@config_blocks list, accumulated routes, lingering Rails::Application
subclasses) that made each subsequent make_basic_app a little
slower. Under Ruby 3.4 + Rails 8.1.3 the per-example time grew 3× over
the run, pushing the full sentry-rails CI past the 15-min timeout.

Hoist make_basic_app to before(:all) and replicate the per-example
bits of Sentry::Rails::Railtie's after_initialize hook in the around
block — re-init Sentry, re-activate tracing / structured logging,
re-register the AJ event handlers. The one-time extensions
(controller methods, streaming reporter, backtrace cleanup, etc.) were
already installed by the initial make_basic_app and persist for the
group.

Also memoize the SidekiqAdapter instance in the :sidekiq context.
Each SidekiqAdapter.new appended to Sidekiq's internal @config_blocks
list and added an on(:quiet) callback; creating a fresh adapter per
example was unnecessary global churn.

Result: spec/active_job goes from 33s → 0.8s, and the full
sentry-rails spec task (Ruby 3.4 + Rails 8.1.3) goes from 9:22 to
2:31 — well under the CI limit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scope reads allowed user keys as symbols, so we gotta symbolize users that
we receive as plain data from the job payload.
@solnic solnic force-pushed the 2933-active-job-tracing-specs branch from d65c086 to 49aa99c Compare June 22, 2026 08:02
@solnic solnic force-pushed the 2587-support-solid-queue branch from d9baea2 to f2c6e34 Compare June 22, 2026 08:02
@solnic solnic force-pushed the 2587-support-solid-queue branch from 0afc95a to c0bc241 Compare June 24, 2026 11:19
@solnic solnic marked this pull request as ready for review June 24, 2026 11:29

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c0bc241. Configure here.

ready = SolidQueue::ReadyExecution.claim("*", 100, process.id)
break if ready.empty? && SolidQueue::ScheduledExecution.none?
ready.each(&:perform)
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drain loop never terminates

Medium Severity

The drain loop only exits when the ready queue is empty and there are no rows in SolidQueue::ScheduledExecution. If scheduled rows exist but none are due yet (scheduled_at still in the future), dispatch_next_batch promotes nothing, claim returns an empty set, and the loop keeps spinning because scheduled rows remain.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c0bc241. Configure here.

SolidQueue::ScheduledExecution.dispatch_next_batch(100)
ready = SolidQueue::ReadyExecution.claim("*", 100, process.id)
break if ready.empty? && SolidQueue::ScheduledExecution.none?
ready.each(&:perform)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drain swallows job exceptions

High Severity

Calling perform on claimed Solid Queue executions returns a result object and rescues job failures instead of re-raising. The shared ActiveJob harness examples expect drain to propagate the same exceptions as ActiveJob::Base.execute, including for expect { drain }.to raise_error cases.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c0bc241. Configure here.

@solnic solnic force-pushed the 2933-active-job-tracing-specs branch 2 times, most recently from 8ed5c5b to 5360bcf Compare June 25, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Solid Queue

1 participant